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Abstract 

With the advent of open-door admissions, the time-honored solution for the 
underprepared college student is to admit and remediate. Because of the tremendous growth in 
college populations in the last thirty years, the number of programs designed to help 
underprepared students has grown significantly. In fact, nearly all senior colleges and all 
community colleges have these programs. Whether or not these programs work or not is still in 
question. In 1968, Roueche reported that college developmental studies program failed the 
students they served. However, a Kulik, Kulik, and Shwalb (1983) meta-analysis found these 
programs to be moderately successful. 

This study used the meta-analysis techniques of Glass (1976) and of Hedges (1985) in 
order to study a subset of developmental studies programs that did not specify the subject matter 
taught. Twenty-seven (27) studies produced 40 effect sizes. The overall effect size was small 
(ES = ,20). The effect size for achievement effects was smaller (ES = . 134, K = 23 ); larger for 
attitude effects (ES = ,27, K = 5); and larger still for persistence effects (ES = .30, K = 12). 
However these averaged effect sizes are all small. 

In these studies where the theoretical underpinnings of the program vyere unknown or 
unreported, college developmental studies programs perform as well as controls but not 
necessarily consistently better than controls. The cause of the poor performance of these 
programs probably is rooted in .their nondescript nature. More study is needed. 
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A Meta-Analysis of the Effects of 
Developmental Studies Programs on 
College Student Achievement, Attitude, and Persistence 

Study Objectives 

It was the purpose of this study to integrate studies of college developmental programs to 
determine their effectiveness on student achievement, attitude, and persistence. Specifically, it 
was the purpose of this study to answer the following questions: What are the effects of 
developmental studies programs on college student achievement? What are the effects of 
developmental studies programs on college student attitude? What are the effects of 
developmental studies programs on college student persistence? The studies chosen for this 
analysis examined only the effects of the total developmental studies program against a non- 
treatment comparison group. The studies selected did not themselves differentiate between the 
subject matter taught. 

Rapid Growth of Developmental Studies Programs and 
Rationale for Study 

Do college developmental studies programs increase the chances of underprepared college 
students' success in college? This question is vital when one considers the increasing numbers of 
students entering these programs and the increasing proportion of college budgets that these 
programs absorb, Moore (1970) reported that 95% of all institutions of higher learning had some 
sort of remedial program. Also, nearly 100% of all community colleges had (and still have) these 
programs (Friedlander, 1980), These numbers represent phenomenal growth. A survey on 



remedial reading found that under 10% of the nations's colleges offered reading in 1960 (Bullock, 
Madden, & Mallery, 1990). By 1984, nearly 84% offered reading (Bullock, Madden, & Mallery, 
1990). Also, by 1980, over 90% of the nation's colleges offered learning skills programs (Henry, 
1986; Wright & Cahalan, 1985). Finally, between 1975 and 1980 remedial enrollment at four 
year institutions increased 72% while total student population increased only 7% (Chang, 1983). 
In one specific instance, one-third of Texas community colleges reported having 75% or more 
students enrolled in remedial mathematics courses (Grable, 1988). 

The most cited research on developmental studies programs is survey research, notably the 
series of Cross studies and series of Roueche et al. studies. Notably, Roueche (1968) reported 
that 90% of students in remedial programs either failed or withdrew from the remedial programs. 
However, a Kulik, Kulik, and Shwalb (1983) meta-analysis of sixty evaluation studies (published 
between 1935 and 1979) of high risk college students reported modest positive effects for 
programs serving high-risk students. This analysis, however, included only nine studies that 
specifically addressed remedial or developmental studies, and only three of these reported positive 
effects on GPA. None of these studies included data from the 1980s. 

Alfred and Lum (1988) noted that the research on developmental studies has been 
deficient in the examination of multiple effects of selected student and institutional variables on 
academic achievement. They called for research that included multiple institutions and examined 
the combined effects of student and institutional variables on academic achievement (Alfred & 
Lum, 1988). Innovative teaching strategies, new classroom management techniques, and new 
classroom technologies have generated a body of research (Schonberger, 1985) that needs to be 
examined, reviewed, and integrated. 
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Limitations of the Study 
This study is limited by the following: 

First, this study is limited to the studies collected by the researcher Though the 
researcher attempted to gather the entire population of studies, such a population is realistically 
unknowable. The inability to obtain some studies because of unavailability or prohibitive costs 
fiarther hampered collection of the population of studies. Finally, these studies did not indicate the 
subject matter taught. 

Second, this study is limited by the accuracy of the reporting of the studies used. 

Third, this sttidy used only one coder, thus subjecting the study to possible bias in the 
coding of each selected study. To lessen this bias, three months after the coding, each 
observation v/as checked for accuracy. This check was later followed by a reliability check of a 
sample of studies. This check showed no coding problems. 

Definitions of both independent and dependent variables, subjects, instalments, and 
experimental designs varied fi-om study to study. This coder used certain global generalizations to 
unify the studies as to definitions, meanings, and characteristics. For example, usually students 
are placed in DSPs because they failed to satisfy the particular requirements for entrance into a 
particular institution's regular curriculum. Finding some objective equalizing factor for the 
differing entrance requirements for all post secondary institutions included in the study would 
have been impossible. Therefore, any study of college programs designed to help students 
prepare for the regular curriculum was included in this meta-analysis. The reader should note that 
the differences between "precollege" students at the Air Force Academy may be very different 
from "precollege" students at an urban community college. Differences in institutions were noted 
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in the coding; however, these results were not included in this study because of the small 
subgroup sizes that developed. These results will be included in a reanalysis of these data. 

Fourth, this study used in its population both weak and strong reports of experiments. 
Weak studies tend to be flawed in some manner, thus lowering the validity and reliability of their 
reported outcomes. The rationale for including these studies is found in the Procedure portion of 
this paper. 

Definitions 

Developmental, remedial, special studies, compensatory, or opportunities programs . For 
the purposes of this study, any program designed to enhance the knowledge, skills, and attitudes 
of students so that they may meet entrance requirement and successfully complete a course of 
study at that institution were classed as developmental. Institutions of higher learning have 
designated these programs to provide instruction in basic reading, writing, and mathematics skills 
for those students who perform below college standards. A more global definition describes these 
programs as " . . . any program, course, or other activity for students lacking the necessary skills 
to perform college work required by the institution" (Cahalan, 1986). Many variables that affect 
the underprepared student qualify this definition including: The kind of high school preparation, 
choice of college, level of college entrance standards, rigor of entry level courses, and availability 
of remedial courses. Varying institutional quality, mission, and definition of "remedial" can 
fiirther complicate a researcher's quest for precise definitions (Cahalan, 1986). In most minds, 
"remedial" has come to mean corrective or curative education that attempts to help students learn 
something that was not previously learned or that has been forgotten (Clowes, 1980). 
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Developmental studies . Developmental studies education emphasizes the building of new 
strengths (Cross, 1976). This includes the enhancement of skills, knowledge, and attitudes that 
may not necessarily be needed to qualify students for more advanced study. Developmental 
services focus on attitude and persistence, and include counseling, advisement, placement, 
tutoring, special classes, and learning labs. 

For the purposes of this project, 'Remedial studies programs" denotes no difference from 
"developmental studies programs." However, "remediation" and "remedial services" will refer to 
corrective acts of remedial mathematics, English composition, and reading/study skills courses. 
These remedial courses can be housed in one division or department of the college, usually called 
"developmental services," or they may be found in traditional departments at colleges that have 
no courses or divisions listed as "developmental." In both cases these courses are often 
interchangeably called "basic," "remedial," or "developmental." 

Compensato^\^ Compensatory education seeks to overcome deprivations associated with the 
home, family, and earlier study (Levine, 1978). Special studies and o pportunities programs tend 
to fall under this definition. These programs tend to be funded by other than the normal funding 
entities. In the literature, all the above terms are used interchangeably (Henry, 1986). 

Methodology 

Glass, McGaw, and Smith (1981) outline the characteristics of meta-analysis as follows: 
First, meta-analysis is quantitative. It uses numbers and statistical methods for organizing and 

I 

extracting information from large masses of data that are nearly incomprehensible by other means. 
Second, meta-analysis does not prejudge research findings in terms of research quality. The 
findings of studies are not ignored a priori by imposing what may be arbitrary and non-empirical 
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criteria of research quality. The influence of study quality on findings is regarded as an a 
posteriori question. Finally, meta-analysis integrates different studies so that general conclusions 
will emerge. 

Procedure 

This study is a quantitative analysis of a sample of studies that address the effects of DSPs 
on students enrolled in higher education. That sample of studies includes articles published in 
journals, dissertations, unpublished works found in ERIC, and published and unpublished 
proceedings of conferences, or yearbooks. 

This section describes procedures for locating studies, quantifying study outcomes, and 
describing study features. The procedure was as follows: 

1. The literature search . First, several indexes to documents and bibliographies were 
searched. The first and most fruitful database was the Educational Resources Information Center 
( ERJC ) index. ERIC provides abstracts of published and unpublished studies, including abstracts 
of papers presented at conferences, progress reports of ongoing research studies, studies 
sponsored by federal research programs, and final reports of educational projects. The second 
database searched was Dissertations Abstracts International ( DAI ). DAI is an index of doctoral 
dissertations submitted from nearly 400 universities. The third was Psyclit . Psyclit is a database 
on CD-ROM that abstracts articles found in psychology journals. The fourth was Sociofile . a 
CD-ROM database that abstrr^ts sociology journals. Fifth, several databases were searched using 
DIALOG Information Services, an on-line database search service. These databases included 
Academic Index , (indexes education journals designed to reach practitioners), Social Sciences 
Citation Index , (indexes articles and lists the citations used in the articles), and Social Science 
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Index , an index of articles in sociology journals. Finally, reference sections of studies selected 
yielded other studies not listed in any of the above databases. 

The descriptors for the various computer searches were as follows: (1) "Higher 
education," (2) "Developmental," "remedial," "underprepared," "high-risk," "at-risk," (3) 
"composition," "reading," "mathematics" "special services," "study skills," and (4) many variations 
of "computer assisted instruction", "audio-tutorials," "auto-tutorials," "Personalized System of 
Instruction (PSI)" and "Keller Plan." 

These searches yielded more than 3000 citations that addressed college DSPs. A reading 
of the abstracts of these studies oi ,he studies themselves, revealed that only about 300 reported 
on quasi-experimental and experimental studies, and the pool was reduced accordingly. Copies of 
these studies were acquired. No predictive or correlational studies with control groups were 
found. 

2. Selection of the studies . The studies were described, classified, and coded. The criteria 
for selecting the studies were borrowed from Kulik, Kulik, and Shwalb, (1983) and modified and 
enhanced for the requirements of this study. The criteria for selection of the studies were as 
follows: (1) studies must have involved students enrolled in college DSPs, (2) studies must have 
reported on measured outcomes for groups of students in "special" programs or groups of 
students being taught using "innovative" teaching methods and must have reported on groups of 
students in comparison programs or in comparison groups receiving traditional instruction, (3) 
when several different articles described the same study, that study was counted only once, using 
the most complete report, (4) when a single article gave results for several semesters, only the 
results for the last semester were included, (5) when a single paper reported only the results for 
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subgroups of students, these results were combined into a composite tally, (6) the study must not 
have made obvious reporting errors. 

3. Coding of the studies . The point of measuring and coding study characteristics to correlate 
study findings with properties of the study. Since the goals of this study were quite broad, the 
coding needed to be comprehensive. Besides the recording of study outcomes, coding involved 
the recording of the "mediating effects" like instructional strategy or study quality. 

Many of the studies reported multiple summary results. Only those deemed most 
important by the coder were entered into the data base. For example, a composition intervention 
may have reported results on three achievement dependent variables-a *^.nal writing sample grade, 
number of words produced on the final writing sample, and number of errors produced on the 
final writing sample. The researcher would have chosen the final writing sample grade. This 
decision is based on the heuristic that reported grades and test scores best describe a study's 
conclusion about a treatment. 

The mediating effects, program characteristics, and summary results were first recorded in 
a data base program called Paradox . This computer program allowed the raw data to be recorded 
directly to a notebook computer via an on-screen form that looked like a paper coding sheet. The 
use of this computer program avoided the need to use perhaps over a thousand sheets of paper for 
coding. Paradox allowed for the sorting and manipulation the data in various ways. For 
example, in studies with multiple effect sizes, recopying program characteristics that did not 
change was an easy affair. This on-line coding sheet helped reduce noise in the data by skipping 
the steps of coding onto a sheet of paper, then entering the data into the computer. After recorded 
in Paradox and checked for accuracy, the data were transferred to a computer spreadsheet called 
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QuattroPro which converted the data int.. American Standard Code for Information Interchange 
(ASCII), a format readable by SAS. QuattroPro also provided an excellent environment for 
making sure the spacing was correct and for checking the details of data set. For example, 
QuattroPro was used to insure that blank spaces that represented missing data were converted to 
dots. 

Since this study used only one coder, what Glass, McGaw, and Smith (198 1) called the 
"principal source of measurement unreliability" (p. 75) (different coders not seeing or judging 
characteristics of a study in the same way) was not a problem. However, to ensure that the 
coding was reliable and that the report is stable, a reliability check was conducted. First, all of the 
studies used were checked for accurate coding. In the check, the coder compared each study to 
the printed output. Changes were made as necessary. Because of missing data in the study 
reports and failure to meet selection criteria, three studies and several observations from other 
studies were omitted. This check was followed by a formal reliability check of !0% of the 
studies. The results indicated no differences with the checked data. Therefore, it was concluded 
that the coding process was reliable. 

4. Measuring study findings . The dependent variables for this statistical analysis 
were the outcomes of the developmental studies program, namely, achievement, attitude, or 
persistence. After the data were recorded, classified, and coded, the summary results were 
mathematically standardized, using effect size as the point estimate. 

5. Analysis . The standard formula for estimating effect size was 

T C 

g= 
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where 



is the treatment mean, 



is the control mean, and 



s 



is either the control standard deviation or a pooled standard 



deviation (Hedges & Olkin, 1985). 

Glass (1976) and Glass, McGaw, and Smith (1981) used the control standard deviation. It 
has the advantage of being uncontaminated by treatment effects (Hedges & Olkin, 1985; Hedges, 
Shymansky, & Woodworth, 1989). 

However, Hedges and Olkin (1985) argued that the most precise estimate of the 
population variance is obtained by pooling, because the assumption of equal population variances 
is reasonable. Hedges and Olkin's (1985) modification of effect size is the same as above except 
in computing the standard deviation with the following formula 



Since (g) has a small sample bias, it is removed by using the (Hedges & Olkin, 1985) 
unbiased estimator of effect size 




where 
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a. Both (g) and (d) (with pooled standard deviations) were calculated when both 
means and stanuard deviations were reported. 

b. Some studies did not provide complete information. If possible, effect sizes 
were reported using alternate methods as discussed by Glass, McGaw, and Smith, (1981). For 
example, exact effect sizes were calculated as follows: 

If (t) was reported and standard deviations are not reported then, 



' 1 
— + — 



\\ T C/ 

However, this formula was not used for studies using matched pairs and where separate standard 
deviations are not reported (Hedges, Shymansky, & Woodworth, 1989). 
If (F) was reported then, 

fF^/t/. 

Many other reported statistics were transformed into effect size estimates via a conversion 
of the reported statistic to a Pearson's (r) (Glass, McGavv, & Smith, 1981). 
The conversion of Person's (r) to (t) is as follows: 



n-2 



as noted in Hinkle, Wiersma, and Jurs (1988). 

c. Next, these effect sizes were then pooled (averaged) to create a common effect 



size 



K 



14 



12 

where K is the number of effect sizes. 

(3) If the sample sizes differ from study to study, then the effect size estimate from large 
studies will be more precise than the estimates from smaller studies. This was controlled by 
weighting the more precise estimates when pooling (Hedges & Olkin, 1985). An approximate 
weight was given by 




r c 



where the pooled weighted estimator is 



w 1 1 k k 



(4) An ANOVA analog ae (Hedges & Olkin, 1985) was performed to determine if all studies and 
the primary groups (reading, mathematics, composition, and total program) shared a common 
effect size for the dependent variables of achievement, persistence, and attitude. The formula for 
the homogeneity statistic ( Q-totaO is 



^ v2 k 



' ' d'" 



i-i at 
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The Hedges and OIkin (1985) process for determining the homogeneity of effect sizes is as 
follows (using the Chi-square table): 

(a) If ( O-total ) is not large or is statistically non-significant at the alpha .05 level, the 
process should be stopped, and it can be concluded that the single effect size fits the data 
adequately. (Note: This analysis adapted a generic SAS meta-analysis program provided by 
Hedges, Shymansky, and Woodworth (1989). In it they use the weighted mean effects in the SAS 
general linear model procedure to produce an QF) statistic. This statistic can be used just as ( Q- 
total) would be used.) 

(b) A significant value of ( O-total ) indicates that the effect sizes are not homogenous. The 
studies are then partitioned into groups by a particularly important dimension or characteristic. 
Since 

the within-class Svdtistics was calculated. 

If ( Q-within ) is nonsignificant, then for class (g) the class effect size was an estimate of the effect 
size of that group. 

(c) A significant ( 0-within ) indicates that the effect sizes were not homogenous within 
classes. The values were examined to help identify classes with especially poor fit. This 
suggested other classifications to observe. 

(d) Thus steps (b and c) are to be repeated until an acceptable level of within class fit is 
obtained or the possible classes are exhausted. 



16 



14 

In accordance with the process noted above, (a) an overall (F) was computed. After that 
(b) the studies were divided into obvious and naturally occurring classes. For example, the 
studies were classified first by instructional methodology, then by subject content taught and 
instructional. The (F) for these subcategories was calculated. Those non-ho ogenous 
subcategories of the studies were divided into those with randomized designs and those without 
randomized designs. If non-randomized studies caused significance, they were removed from 
further study. 

Two of the above analyses were done. The first used averaged effects within the study so 
that no matter how many effects sizes were coded, only one achievement, attitude, and/or 
persistence effect size from each study would be analyzed. This method protects the 
independence of the data. 
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Results 

Forty effect sizes were produced from the 27 studies collected. These the overall effect 
for developmental studies programs was positive; the overall effect size was a small .20. Even 
though 60% of these effect sizes were positive, strong negative effects of remedial programs' 
achievement effects significantly moderated the overall effect size (See Figure 1 and Table 1). 

When achievement effects were considered, the percentage of positive effect sizes 
dropped to 52% of the total. Studies that reported achievement results, post tests, final course 
grades, next course grades, or GPA, had the majority of negative effect sizes. The overall effect 
size for student achievement was .134. The majority of these studies did not specify a particular 
methodology other than a developmental studies program; therefore, they were categorized as 
simply "remediation" (See Table 1). The overall effect size for student achievement-remediation 
was ,07. Other classes include individualized (ES == -.50), learning assistance center (ES == -.07), 
block (.33), and a group of studies that examined counseling (ES == ,75) 

Only five studies considered student attitude for total remedial programs. The overall 
effect size was .27 (See Table 2). Twelve studies considered student persistence (See Table 3); 
the overall effect size for these studies was ,30. The persistence statistics are all effect size 
estimates that are derived from percentages of students who returned to the college the final 
semester of the treatment included in the respective: studies. These percentages were converted to 
Chi-square then transformed as noted in the procedure section of this paper. About 75% of the 
studies report positive return rates for students. 

Finally, the weighted ANOVA analogue did not show any statistical significance. 
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Figure L Stem-and-leaf display of effect sizes from studies of developmental studies programs 
that did not report differences by the subject matter taught. These studies reported the effect of 
the total developmental studies program on the student. 
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Table 1 
Classification of Studies 
Student Achievement for Total Program 
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Table 2 
Classification of Studies 
Student Attitude for Total Program 
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Table 3 
Classification of Studies 
Student Persistence for Total Program 
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Discussion and Conclusions 

The statistic used for comparison purposes was effect size. Effect size is a quantitative 
way of describing how well the average student who received the intervention performed relative 
to the average student who did not receive the intervention (Borg & Gall, 1989). A 0 effect size 
means that an average student receiving the innovative instructional methodology did no better or 
worse that an average student not receiving the innovative methodology, while a positive effect 
size means that the average student receiving the intervention did better than the average student 
not receiving the intervention. For example, an effect size of .20 means that the average student 
in the experimental methodology scored one-fifth of a standard deviation higher than the average 
student in the control group. This would place that student at the 58th percentile in the control 
group, "Researchers, " according to Borg and Gall (1989), "consider effect sizes larger than ,33 
to have practical significance , that is, the effect is large enough to make a worthwhile difference in 
the outcome" (p. 7). However, Hinkle, Wiersma, and Jurs(1988), contrast Cohen's statement 
that an effect size of ,25 was small with Feldt's conclusion that an effect size of .20 was small. 
They conclude, along with Cook and Campbell (cited in Hinkle, Wiersma, & Jurs, 1988), that 
consciousness of an appropriate effect size magnitude is a complex and perplexing issue. For the 
broad purposes of this study, determining practical significance of effect size involved 
considerations of the number of effects in the subcategory of studies, the nature of the 
subcategory itself, and mediating factors that may be in the studies. Therefore, effect sizes were 
compared directly-with caution. For instance, this researcher saw an instaictional methodology 
requiring the latest technology, extra personnel, and extra staff development as having to have a 
higher mean effect size in order to have equal practical significance with programs, for example. 
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that only require student scheduling changes. Practical significance requires an understanding of 
tlie fact that the context of the effect size is just as important as the magnitude of the effect size. 

With that in mind, it is difficult to conclude that developmental programs are a cure-all for 
the problems that underprepared freshman face. This researcher would conclude that the small 
effect size for achievement (ES = . 1 34) indicates no pr actically significant difference between the 
students in developmental programs and similar students in control groups-using the above 
discussion as a guide. However, these studies are a subset of all of the studies of developmental 
studies programs. Before the fact, these studies are homogenous in that these studies do not 
delineate the different subjects taught in the remedial program. For example, these studies 
basically average together English, reading, and math course scores. Average English and reading 
scores were no doubt severely moderated by lower math scores. This conclusion is partially 
supported by the larger effect size for persistence studies. Students may be failing math but may 
also be finding success in English and reading courses, and that fact is keeping them enrolled. 

The problem then appears to be partly in the selection of the studies. What needs to 
follow this study is a series of studies that collect and integrate studies that are classified by 
subject area taught. Four such meta-analyses are currently works in progress by this author. 

The rest of the problem seems to be in the developmental studies programs themselves. 
Of the 23 studies that reported effects on student achievement, 15 had effects sizes that ranged 
from -.32 to .57. Quite a few studies reported small and moderate effects for their programs. 

Clearly there is a need for these programs. There is an even clearer need that these 
programs ground what it is they do in something other than providing watered-down versions of 
regular courses; that is, to simply provide remediation is not enough to help underprepared 



24 



22 

college students. Programs should have strong theoretical bases, like counseling and the block 
programs, which seemed to consistently produce moderate to strong positive effect sizes. 
Because of the changing demographics of the college student and because of ever-increasing 
percentages of underprepared students, these programs simply must work better. 
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